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ABSTRACT 

Raytheon’s  Infrared  Operations  (RIO)  has  invented  and  developed  a  new  type  of  focal  plane  array 
with  “Image  Processing  on-the-chip”  named  the  Adaptive  IR  Sensor  (AIRS).  The  AIRS  FPA  is  based 
upon  the  human  retina  in  that  it  performs  signal  processing  near  the  photoreceptors.  The  AIRS  FPA  has 
been  reduced  to  practice  and  adaptively  removes  detector  and  optic  temperature  drift  and  1/f  induced 
fixed  pattern  noise.  This  3rd-generation  multi-mode  IRFPA,  also  called  a  Smart  FPA,  is  a  256x256-array 
format  capable  of  operation  in  four  modes:  1)  Direct  Injection  (DI),  2)  Adaptive  Non-uniformity 
Correction  (NUC),  3)  Motion/Edge  Detection,  and  4)  Subframe  Averaging.  Nova  Research  has 
developed  a  Modular  IR  Application  Development  System  (MIRIADS)  which  is  a  compact  single  board 
camera  system  that  is  highly  integrated  with  the  dewar  assembly.  The  MIRIADS  system  coupled  with 
the  AIRS  Smart  FPA  results  in  a  very  high  performance  wide  field  of  view  IR  Sensor  and  processing 
system  with  integrated  in  one  of  the  smallest  packages  to  date. 


1.  INTRODUCTION  TO  BIOLOGICALLY  INSPIRED  CIRCUIT  ARCHITECTURES 

Raytheons  Infrared  Operation  has  designed  ROICs  and  IRFPAs  that  make  up  for  the  defects  in  IR 
Detectors  and  also  continue  to  advance  imaging  technology  to  the  point  where  a  Sensor  could  be 
integrated  onto  a  single  chip.  Scientists  at  RIO  looked  towards  biological  sensors  for  answers  on  how  to 
conduct  signal  processing  on  the  chip  and  at  the  same  time,  solve  problems  of  pixel  uniformity  and 
operability.  Figure  1  is  a  schematic  of  the  Human  eye  with  retinal  photoreceptors  and  processing  cells. 
Figure  2  shows  a  flow  diagram  of  the  visual  pathway  and  associated  processing  at  each  level,  from  M.M. 
Gupta  and  G.K.  Knopf1. 

Our  first  goal  was  to  design  circuitry  with  both  temporal  filtering  (AIRS)  and  spatial  processing 
(TAIP)  at  the  pixel  level  in  order  to  present  a  more  uniform  preprocessed  image  to  the  image  processor. 
As  figure  2  illustrates,  performing  both  spatial  and  temporal  filtering  at  the  detector  level  has  obvious 
benefits.  Since  the  computation  modules  required  for  filtering  are  often  simple  convolutions,  performing 
these  2-D  functions  on  the  image  plane  requires  only  neighborhood  interconnections  with  the  weights 
controlled  synaptically.  Attempting  to  do  this  in  a  serial  digital  computer  is  far  more  complex  and 
involves  large  amounts  of  memory,  memory  accesses,  multiplies,  and  adds. 
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la.  Side  look  at  the  components  of  the  human  eye 
reveals  a  superior  adaptable  WFOV  vision 
system. 


lb.  The  human  retina  has  a  densely  packed  set  of 
rods  and  cones,  and  4  adjacent  layers  of  image  pre¬ 
processing. 


The  human  retina  is  backside  illuminated  like  many  IRFPA  designs.  There  is  a  lot  of  processing  that 
goes  on  in  the  human  retina  that  is  the  subject  of  Research.  The  AIRS  chip  performs  temporal 
processing  similar  to  the  bipolar  cells,  but  how  closely  the  AIRS  type  processing  replicates  the  retina  is 
the  subject  of  future  research.  Suffice  it  to  say,  AIRS  is  a  stand  alone  integrated  image  processor  on  a 
chip. 
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Figure  2.  Biological  vision  research  indicates  that  temporal 
and  spatial  processing  is  performed  near  the  retina  for  the 
purposes  of  noise  reduction  and  information  enhancement. 


In  the  front  end  of  a  2-D  imaging  sensor  using  massively  parallel  processing  elements,  these 
convolutions  can  be  performed  in  the  equivalent  of  a  single  clock  cycle.  Thus  the  end  goal  of  Smart 


FPAs  is  to  mimic  biological  sensors  and  to  leverage  advances  in  submicron  ROIC  designs  in  order  to 
enable  smaller  and  smarter  IR  detection  systems. 

The  full  up  MIRIADS  system  is  shown  in  figure  3.  Although  the  MIRIADS  is  not  approaching  the 
optimization  of  the  near  perfect  eye,  it  has  a  high  level  of  integration  with  more  processing  functions 
being  performed  near  the  focal  plane  array  than  most  IR  or  Visible  sensor  systems.  The  processing  that 
is  done  on  the  FPA  is  Subframe  Averaging  and  Temporal  Filtering,  while  the  near  Focal  Plane  MIRIADS 
electronics  performance  timing  and  control  signals,  digitization  and  a  set  of  parallel  connections  via  the 


Figure  3.  Photo  of  the  MIRIADS  system  with  the  cover  off.  Fisheye  Lens,  Dewar,  FPA,  A/C,  timing 
and  bias  generation  and  image  processing  all  on  one  card.  While  the  MIRIADS  is  not  as  compact  or 
mature  as  the  eye  we  are  headed  that  direction. 


2.  ADAPTIVE  IR  SENSOR  DESIGN  OBJECTIVES  AND  PERFORMANCE  RESULTS 

The  Adaptive  Infrared  Sensor  (AIRS)  FPA  readout  IC  was  developed  with  support  from  IRAD  and 
NVESD.  The  technical  objectives  of  this  3rd  Generation  design  was  to  utilize  Readout  Integrated  Circuit 
(ROIC)  advancements  to  demonstrate  biologically  inspired  temporal  filtering  at  the  pixel  to  improve  the 
overall  noise  performance  of  focal  plane  array.  Over  the  past  decade  both  DOD  and  industry  have  made 
large  investments  into  infrared  improvements,  and  these  detectors  have  been  optimized  for  operation  near 
the  theoretical  limits.  Detector  fabrication  has  also  been  systematically  improved  with  current  fabrication 
giving  high  yields  and  low  process  defects.  A  number  of  successful  research,  development,  and 
production  programs  have  advanced  and  sustained  a  high  level  of  technical  excellence  involving  both 
photovoltaic  HgCdTe  and  InSb  detectors.  However  even  with  the  successes  in  making  better  IR 
Detectors  we  are  asymptotically  approaching  an  upper  limit  on  the  performance  and  yields  of  Detectors. 
Therefore  the  ROIC  is  deemed  to  be  the  primary  FPA  component  capable  of  yielding  significant 
improvements  in  performance. 

The  ROIC  is  compatible  with  either  detective  material,  and  serves  to  reduce  fixed  pattern  noise  on  the 
FPA  rather  than  rely  on  warm  external  electronics  for  this  vital  function.  This  breakthrough  advancement 
in  on-FPA  processing  will  enable  the  future  development  of  smaller,  lighter,  and  lower  power  systems 
with  ever  greater  operational  capabilities  than  available  in  today’s  products.  Additionally,  the  AIRS  is 
the  first  ROIC  to  demonstrate  an  ability  to  adaptively  “fix”  bad  pixels  on  the  focal  plane,  thus  improving 
the  offset  nonuniformity  coming  off  the  FPA.  This  capability  also  increases  the  useful  yield  and  lowers 


cost  of  the  detector  arrays  composing  the  focal  plane.  To  demonstrate  the  ability  to  fix  pixels,  an 
engineering  grade  LWIR  30  um  256  x  256  detector  array  from  another  effort  was  used  on  the  AIRS  R&D 
lot. 


Performance  breakthroughs  of  the  AIRS  Smart  FPA  have  been  demonstrated  in  the  following  four 
modes  of  operation. 

•  Direct  Injection  (DI).  The  AIRS  operates  as  a  standard  2nd  Generation  imaging  FPA. 

•  Adaptive  THP  Scene  Based  Non-Uniformity  Correction.  This  mode  allows  adaptive  reduction 
of  the  Low  frequency  noise  and  drift  component  from  the  scene.  The  fixed  pattern  noise  (FPN)  is 
significantly  reduced  in  the  temporal  high  pass  filter  mode(THPF).  The  THPF  lowers  the 
temporal  noise  making  marginal  pixels  (i.e.,  low  responsivity  or  high  noise)  perform  sufficiently 
for  use.  The  number  of  outliers  (speckles)  are  thus  effectively  reduced,  resulting  in  a  dramatic 
improvement  in  operability. 

•  Subframe  Averaging.  This  mode  offers  a  tremendous  advantage  over  existing  designs.  By 
averaging  subframes,  high  photon  shot  noise  is  reduced  and  sensitivity  is  improved.  Using  low 
noise  electronics  and  a  state  of  the  art  A/D  converter  vastly  improved  sensitivity  of  3  x  in  NEDT 
in  the  subframe  accumulation  mode  is  possible. 

•  Motion  Detection/Edge  Enhancement:  This  mode  offers  the  ability  to  remove  all  stationary 
scene  clutter  and  only  detect  moving  targets  and  edges.  This  mode  is  the  THP  mode  running  at  a 
very  high  update  rate. 

An  overview  of  the  measured  performance  of  the  Adaptive  IR  Sensor  is  shown  in  table  1 . 


Table  1.  Summary  of  the  Adaptive  IR  Sensor  performance  results. 


Performance  Parameter 

Mode 

Direct  Injection 

THPF/  Motion  Detection 

Subframe  Averaging 

Array  Format 

256  x  256 

256  x  256 

256  x  256 

Pixel  Pitch 

30  um 

30  um 

30  um 

Spectral  Passband 

LWIR 

LWIR 

LWIR 

Flux  Level  @Tbb=  295  K, 

F/2 

3.5  xIO1'  Ph/cm^-sec 

3.5  xIO1'  Ph/cm^-sec 

3.5  xIO1'  Ph/cm^-sec 

#  of  Outputs 

1 

1 

1 

Frame  Rate 

>  100  Hz 

>  100  Hz 

20-100  Hz 

Output  Voltage  Swing 

>  3  Volts 

>  3  Volts 

3 

Instantaneous  Dynamic 
Range 

>3500 

>3500 

>4800 

Integration  Time 

47  usee 

36.5  msec 

36.5  msec  x  Nsub 

Full  Well  Capacity 

12  Me- 

6  Me- 

Averages  100  frames 

Integrated  Charge  (e-) 

6.5  M  e- 

4  M  e- 

Averaged  400  M  e- 

%  of  Full  Well 

54% 

66% 

54% 

Mean  NEDT  (  BLIP) 

26  mK  (20  mK) 

<  28  mK  (23  mK) 

<  20  mK 

Figure  4  shows  the  output  video  skyline  plot  of  all  256  x  256  pixels  in  Direct  Injection  mode  and 
Temporal  High  Pass  mode.  The  sub- specification  “C”  grade  detector  exhibits  a  localized  defect  of  about 
100  pixels  on  the  right  part  of  the  skyline  that  reduces  the  operability  in  the  Direct  Inject  mode  to  99.8  %. 
In  the  Temporal  High  Pass  mode  the  array  uniformity  is  reduced  from  40  mv  to  14  mv.  Also  notice  the 
group  of  bad  pixels  (Fig  4a)  have  been  “fixed”  or  brought  into  the  mean  of  the  distribution  (Fig  4b). 
These  are  the  first  ever  demonstrated  results  of  actual  bad  pixel  correction  performed  in  the  ROIC,  with 
the  FPA  operability  approaching  100%. 


A  key  issue  of  the  On-FPA  Adaptive  NUC  is  the  ability  to  maintain  low  residual  Fixed  Pattern  Noise 
(FPN)  during  normal  operation.  Past  Smart  FPA  efforts  have  discussed  the  need  for  on  FPA  processing 
and  the  need  for  On-the-FPA  NUC2,3.  We  performed  a  series  of  tests  comparing  the  FPN  of  the  Direct 
Injection  Mode  of  AIRS  with  the  Temporal  High  Pass  mode. 


Figure  4a.  Direct  Injection  Mode  skyline  plot  Figure  4b.  Temporal  High  Pass  skyline  plot  shows 
shows  good  raw  offset  uniformity  (~40  mv  la,  excellent  raw  Offset  Uniformity  (14  mV 

a/p=  2.5%)  la,  a/|Li<l%)  and  Bad  Pixel  Substitution 

In  the  DI  mode  a  standard  2  Point  NUC  was  performed  in  the  Electronics.  In  the  THPF  mode,  a  1- 
point  Offset  NUC  was  performed  to  remove  the  14  mv  noise  that  originates  mainly  in  the  column  amp 
and  unit  cell  buffers.  Figure  5  illustrates  the  ability  of  the  AIRS  FPA  to  maintain  very  low  residual  fixed 
pattern  noise  over  scene  temperature  excursions  (Fig  5a)  and  time  (Fig  5b). 
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Figure  5a.  Residual  Spatial  Noise  in  Temporal 
High  Pass  mode  vs.  Direct  Injection  mode  shows 
the  ability  of  the  THPF  to  maintain  FPN  near  the 
temporal  noise  floor. 


Figure  5b.  Long  term  stability  of  Temporal  High 
Pass  mode  vs.  Direct  Injection  shows  that  the 
THPF  maintains  the  fixed  pattern  noise  near  the 
temporal  noise  during  a  5  hour  stability  test. 


During  performance  testing,  residual  spatial  noise  of  the  THP  mode  increased  only  15%  over  a  5  hour 
period.  This  result  illustrates  the  robustness  of  the  on-FPA  techniques  to  maintain  low  FPN  and  stability 
over  time. 

Imaging  tests  were  performed  in  all  modes  of  operation.  Figures  6,  7,  and  8  illustrate  the  DI,  THP, 
and  Motion  Detection  mode,  respectively. 


Figure  6.  AIRS  FPA  exhibits  good  imagery  and  operability  in  Direction  Inject  Mode. 


Figure  7.  Images  from  the  THPF  mode  test.  When  the 
THPF  is  running  a  slow  update  rate,  very  few  image  artifacts  exist. 


Figure  8.  Sequential  images  taken  in  the  Moving  Target/Edge  Enhancement  mode.  The  FPA 
performs  edge  enhancement  on  moving  targets  and  creates  Mach  bands  on  the  hot  target. 


2.  AIRS  INTEGRATED  WITH  MIRIADS 


Figure  9  shows  an  early  conceptual  representation  of  the  MIRIADS  camera  system.  A  unique  feature  of 
the  system  is  that  the  single  circuit  board  (the  motherboard,  9.0  inches  in  diameter)  that  is  required  to 
operate  the  FPA  actually  penetrates  the  vacuum  space  of  the  integral  vacuum  dewar  and  is  in  fact  an 


important  part  of  the  vacuum  integrity  of  the  system.  As  a  technology  demonstration  effort,  this  project 
had  a  goal  of  demonstrating  that  a  cryogenically-cooled  infrared  FPA  in  an  evacuated  dewar  can  be 
mounted  on  the  same  board  as  the  warm  electronics  that  operates  it.  In  this  process,  electrical  vacuum 
feed-thru  connections  normally  required  of  these  systems  are  eliminated;  they  are  replaced  by  internal 
signal  layers  of  the  multi-layer  printed  circuit  board.  Another  benefit  of  this  design  is  that  because  of  the 
very  short  distance  between  the  FPA  output(s)  and  the  associated  preamplification  and  digitization 
circuitry,  the  signal  capacitance  is  very  low;  this  permits  very  high  frequency  operation  for  advanced 
imagers  of  the  future. 
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Figure  9.  A  single  circuit  board  (the  motherboard)  is  required  to  operate  the  FPA,  and  additional 
processing  daughterboards  may  be  used  to  post-process  the  real-time  image  data. 


In  addition,  the  MIRIADS  system  architecture  permits  a  common  datapath  to  be  used  to  send  real-time 
video  and  control  data  from  the  motherboard  to  a  variety  of  processing  daughterboards  as  shown  in 
Figure  1.  This  is  also  indicated  in  Figure  10  in  which  numerous  identical  boards  with  different 
programmable  logic  implementations  may  be  used  to  process  the  real-time  FPA  image  data.  A  more 
detailed  discussion  of  the  MIRIADS  is  given  in  another  paper4. 


To  be  as  flexible  as  possible,  the  system  was  designed  to  permit  cooling  of  the  FPA  with  the  use  of  liquid 
nitrogen  (LN2)  or  an  electric  cryocooler  unit  through  the  cooling  port  located  in  the  lower  dewar 
assembly  as  shown  on  the  left  side  of  Figure  10.  As  will  be  shown  later,  this  program  also  developed  a 
large-capacity  LN2  reservoir  dewar  subsystem  that  mates  to  the  cooling  port  that  provides  many  hours  of 
cooling  capacity. 


A  design  goal  was  to  produce  a  system  that  provided  an  easy  interface  to  a  commercially-available  image 
capture  and  display  system  so  as  not  to  divert  from  the  main  goal  of  producing  a  highly  modular, 
configurable  system.  For  this  reason,  the  SE-IR  Corporation’s  “CamIRa”  display  frame  grabber,  DSP 
board  and  software  was  identified  and  used  to  provide  real-time  display  and  digital  data  storage 
functions. 
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Figure  10.  Schematic  view  of  the  MIRIADS  camera  system  indicating  that  the  motherboard  is  an  integral 

part  of  the  vacuum  integrity  of  the  cryogenic  dewar. 


Figure  1 1 .  Three  views  of  an  existing  MIRIADS  system  operating  the  Neuromorphic  Infrared  focal 
plane  array  showing  a  six-inch  scale  for  size  comparison. 


Figure  11  shows  the  single  data  cable  that  is  used  to  deliver  the  16-bit  differential  data,  pixel,  frame  and 
line  clocks  to  the  display  system.  This  cable  also  permits  software-generated  camera  control  functions  to 
be  performed  (i.e.,  integration  time,  frame  rate  or  other  camera  control  parameters). 

It  became  clear  through  the  initial  operation  of  the  MIRIADS  that  the  small  volume  of  LN2  that  could  be 
poured  into  the  cooling  port  would  only  keep  the  detector  cold  for  approximately  ten  minutes.  The  heat 
load  of  the  system  was  previously  measured  to  be  approximately  1  watt  at  77K,  low  enough  for  a  cryo- 
cooler,  but  high  enough  to  require  multiple  milliliters  of  LN2  if  poured  directly  into  the  cooling  port. 
The  large  capacity  LN2  dewar  shown  in  Figure  4  was  developed  to  provide  many  hours  of  cooling  while 
horizontally  supporting  the  MIRIADS  camera  system  to  make  it  useful  in  data  collection  operations. 


Figure  12.  A  high-capacity  LN2  dewar  system  was  developed  to  provide  long-term  cooling  and  a 
horizontal  mounting  for  the  MIRIADS  camera  system. 

The  motherboard  was  designed  with  “in  system  programmable”  (ISP)  complex  programmable  logic 
devices  (CPLDs)  that  permit  the  reprogramming  of  virtually  any  set  of  clock  patterns  required  by  the 
particular  FPA  for  its  operation.  In  addition,  flexible  clock  level- shifting  circuitry  was  included  to 
supply  both  positive  and  negative  clock  rails.  A  variety  of  low-noise  adjustable  FPA  bias  generators  are 
included  to  accommodate  the  power  and  bias  requirements  of  virtually  all  FPAs  that  currently  exist.  The 
motherboard  includes  four  channels  of  gain-  and  offset-adjustable  preamplification  circuits,  four  14-bit 
A/D  converters  and  on-board  digital  multiplexers  to  produce  the  digital  data  stream  that  is  driven  off  the 
board  through  either  RS-422  or  LVDS  differential  drivers  to  provide  high  noise  immunity  in  harsh  EMI 
environments,  system  complete. 


4.  FUTURE  APPLICATIONS 

Figure  13  gives  an  example  of  next  generation  IRFPA  insertion  using  an  even  more  compact  version 
of  a  MIRIADS  like  system.  In  order  to  achieve  these  results,  we  endeavor  to  continue  to  look  to  nature 
to  design  sensors  that  are  biologically  inspired4"5  and  modeled  after  the  Retina  as  well  as  take  advantage 
of  recent  advanced  in  ASIC  and  EPLD  design. 
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•  Bolometer  IRFPA 

•  <  6  grams  goal 

•  On  FPA  Processing 
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•  Combat  ID/Battlefield  Recon 

•  Urban  Warfare 

•  Intelligence  Gathering 

•  Special  Target  Surveillance 


Figure  13.  Example  of  how  3rd  Generation  sensor  could  enable 
high  performance  mission  capability  for  Micro  UAVs. 

Using  the  biologically  inspired  AIRS  and  TAIP  technology,  next  generation  IRFPAs  can  provide 
benefits  in  numerous  21st  century  missions: 

•  Unattended  Sensors 

•  Manportable  Sensors 

•  Micro  UAVs 

•  Ground/W all  Penetration 

•  Missile  Warning 

•  Missile  Seekers 

•  Multispectral  Surveillance 

•  Advanced  Large  Area  FPAs 


5.  SUMMARY 

Raytheon  has  demonstrated  the  ability  to  do  processing  on  the  FPA  that  has  previously  been  done  in 
the  FPA  warm  electronics.  DOD  funding  has  resulted  in  dramatic  improvement  in  both  MWIR  and 
LWIR  detectors.  However,  the  IR  detector  industry  is  asymptotically  approaching  the  upper  limit  of 
performance  and  operability.  What  is  needed  now  are  advanced  ROICs  that  actually  improve  the 
performance  of  detectors  and  increase  useful  operability  of  FPAs.  With  the  requirement  for  smaller, 
smarter,  and  lower  power  sensors,  more  image  processing  needs  to  be  moved  onto  the  FPA.  To  that  end 
Raytheon  has  demonstrated  biologically  inspired  massively  parallel  processing  of  images  at  the  pixel 
level  with  excellent  first  pass  success.  Importantly,  the  AIRS  FPA  requires  only  3  biases  and  4  clocks  to 
operate.  Raytheon’s  Smart  FPA  efforts  have  demonstrated  significant  breakthroughs: 

•  Biologically  inspired  Parallel  Processing  at  or  near  the  pixel  level 

•  Industry  first  IRFPA  to  “Fix”  bad  pixels  on  the  FPA 


•  Excellent  uniformity  of  <  1%  sigma/mean  with  LWIR  detectors 

•  Superior  Long  Term  stability  and  image  quality 

•  AIRS  Multifunctional  FPA:  DI,  Adaptive  NUC,  Subframe  Averaging,  and  Motion  Detection 

•  Enables  a  higher  level  of  integration  and  sophistication  of  image  processing  on  or  near  the  FPA 
for  a  total  reduction  in  system  size  while  actually  increasing  functionality. 

The  excellent  first  results  achieved  with  Raytheon’s  Adaptive  IR  Sensor  and  TAIP  have  proven  that 
Multifunctional  3rd  Generation  FPAs  with  on  the  FPA  processing  are  both  possible  to  fabricate  and  offer 
more  capability  than  existing  2nd  generation  IRFPAs. 
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